DOCUHERT BESOBE 



BD 128 469 



TH 005 664 



EDBS PBICE 
DESCBIPTOBS 



iOTHOB Fink, Arlene 

TITLE The Use of Data from Competency-Based Measurement: An 

Instructional Developer's view. 
PDB DATE [Apr 76] 

HOTE 1«p.: Paper presented at the Annual fleeting of the 

N^^tional Council on Heasurement in Education (San 
Franciscor Calif ornia, April 19-23, 1976) 

HF-$0,83 HC-$1.67 Plus Postage, . 
*Achievement Tests; *Critorion Beferenced Tests; 
♦Instructional Materials; *l!aterlal Development; 
♦Performance Based Education; Performance Tests; 
♦Test Construction; Test Eeliability; Test Validity; 
Validity 

ABSTBACT ' 

Attention has recently been focused on the need to 
design and validate instructional materials that guarantee competent 
student performance. In preparing such products, developers have come 
to rely on competency-based achievement tests. Unfortunately , 
expertise for constructing reliable and valid competency-based 
instruments is not currently available in a form than an 
instructional developer can readily translate into practice. The 
formulation of objectives to guide instrument construction is 
problematic for the developer because few dependable rules exist to 
help select and state them. Formulating items is also difficult 
because many developers lack training in test writing. Further, major 
issues arise when the developer must establish the meaningfulness of 
the scores that result from competency-based instruments, a difficult 
and costly task that requires psychometric skill and a generous 
budget* Finally, a discerning developer knows in advance of 
instrument validation that little assistance will be available from 
psychometricians who have long been arguing over the designs and 
statistics to be used validating competency-based tests. Data derived 
from competency-based instruments must therefore be used with caution 
and confirmed with additional sources of information like 
observations v^nd interviews. Developers should also require test 
construction and psychometric expertise. (Author) 



♦ Documents acquired by EBIC include many informal unpublished ♦ 

♦ materials not available from other sources. EBIC makes every effort ♦ 

♦ to obtain the best copy available. Nevertheless, items of marginal ♦ 

♦ reproducibility are often encountered and this affects the quality ♦ 

♦ of the microfiche and hardcopy reproductions EBIC makes available ♦ 

♦ via the EBIC Document Eeproduction Service (EDBS). EDRS is not ♦ 

♦ responsible for the quality of the original document. Beproduct ions ♦ 

♦ supplied by EDBS are the best that can be made from the original. ♦ 



EKLC 



oo 
rsi 



o 



The Use of Data from 
Competency-Based Measurement: 
An instructional Developer's View 
Arlene Fink, Ph.D. 
Center for the Study of Evaluation 
UCLA Graduate School of Education 
Los Angeles, California 



U.S. OCPAftTMEXr OF HeALTH 
"WCATIONAWELFAHE * 
NATIONAL INSTtTUTg OF 
EDUCATION 

^ ?HE PERsSn np Xot^ RECEIVED FROM 

■ <^I?pn^,^^°"'^5 V'EW OR OPl^i^,y\ 
lltl^P? NECESSARILY REPRE 

SENT OPPICtAL NATIONAL I NSTI tVjtc > 
EDUCATION POSITION OR POUCV 



CO 

to 

CD A Paper presented to the National Council on Measurement In Education 

San Francfsco, California 
April 21, 1976 



Competency-Based 
I 

ABSTRACT 

Attention has recently been focused on the need to desfgn and 
Validate Instructional materials that guarantee competent 
student performance, fn preparing such products, developers have 
come to rely on competency-based achievement tests. Unfortunately, 
expertise for constructing reliable and valid competency-based 
Instruments Is not currently availble in a form that an InJStruct ional 
developer can readily translate into practice. The formulation 
of objectives to guide Instrument construction Is problematic for 
the developer because few dependable rules exist to help select 
and state them. Formulating items Is also difficult because many 
developers lack training In test writing. Further, major Issues 
arise when the developer must establish the meanlngfulness of the 
scores that result from competency-based instruments, a difficult 
and costly task that requires psychometric skill and a generous 
budget. Finally, a discern tng developer knows In advance of Instru- 
ment validation that little assistance will be availble from psycho- 
metricians who have long been arguing over the designs and statistics 
to be used when validating competency-based tests. Data derived 
from competency-based instruments must therefore be used with 
.caution and confirmed with additional sources of Information like 
observations and Interviews. Developers should also acquire test 
cdnstructlon and psychometric expertise. 
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In recent years, professional educators have focused attention on 
the creation and empirical validation of Instructional materials and pro- 
cedures. To them, the words "emtrlcally validated'' usually mean a guarantee 
that when systematically developed materials are used as Intended, students 
will acquire Important knowledge and skills and be able to competently 
perform educationally significant tasks. The Idea that instructional 
products can and should ensure learner competence has been buttressed 
by the movement for performance-based or competency- based education and 
measurement, a relatively recent Innovation that Is thought to offer signifi- 
cant mechanisms for educational reform (Hooc & Blackwell, 1975), 

There are, unfortunately, numerous definitions of competence, and they 
vary greatly. Schalock . nd Thomas (1973) have made a useful distinction be- 
tween the two most common meenings of the term. Competence, they suggest, 
can be equated with the mastery of knowledge and skills assumed to be neces- 
sary to perform a particular job, or it can refer to demonstrated ability to 
bring about outcomes specific to a given job description. There are few 
school' situations In which instructional product developers can follow their 
students to their jobs to observe their performance. Thus, the most frequent- 
ly used definitions of competence in instructional development are those re- 
lated to the provision and measurement of knowledge and skills. 

The procedures that should be used to empirically validate competency- 
based Instructional materials have beer described In the Instructional pro- 
duct development literature (Baker, 1973; Markle, 1967) « Although develop- 
ment experts may disagree on the specifics of instructional design, they are 
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united in their acceptance of the need to conduct systematfc field trials 
of the product with a representative sample of the intended learners. They 
also advocate, at a minimum, the use of data from achievement tests that mea- 
sure the product's effectiveness in attaining its objectives and promoting 
competence. 

The Center for the Study of Evaluation (CSE) at UCLA is engaged in the 
development of numerous competency-based instructional products to train eval- 
uators to organize and conduct evaluations. Although attempts are made 
through special studies to follow performance on the job, most development 
activities are guided by data derived f rom achievenjent tests that assess 
mastery of knowledge and slcills. The data are used to answer questions con- 
cerning how well and how much students are learning in order to improve the 
product so that their competence is maximized. Thus, implicit in the design 
of CSE's products is an acceptance of the view that competency-based educa- 
tion serves its potential audience well if it provides them with the skills 
and knowledge presumed to be related to successful performance, and the belief 
that expertise exists and is available to develop reliable and valid measure- 
ments of the extent to which the skill and knowledge are acquired. It is the 
latter belief--that expertise is available for developing reliable and valid 
competency-based measurement devices like achievement tests--which this paper 
explores from the point of view of one of CSE's instructional developers. 
To do this, competency-based measurement in instructional product develop- 
ment will be defined, and issues in the development and validation of com- 
petency-based achievement tests will be discussed. 
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Competency-Based Achievement Tests 
and Instruct tonal Product Development 
Competency-based achievement tests are designed to provide a measure 
of the extent to which an ,structional product's objectives have been achieved 
In equipping learners witn specific competencies J They have three essential 
charact'^T-i sties: 

1. they are based on clearly-defined performance objectives representing 
competencies ; 

2. test Items are specifically designed to measure the objectives; 

3. scores are. interpreted in terms of attainment of a pre-set 
criterion or standard of performance." 

Formu 1 a t i ng Ob J ect i ves/Competenc i es 

One of the instructional product developer's basic responsibilities is 
to formulate the performance objectives that frame the product and guide 
achievement test development. In competency-based education, this Is usually 
done by obtaining the advice of experts, analyzing the subject area to be 
learned, conducting a needs assessment, and by monitoring student progress 
during field tests of the product. Several problems invariably arise when 
formulating objectives. The most common include determining which ones to 
select; whether to state them In specific or genes'al terms; how to sequence 

them (from simple to complex? from abstract to concref:e7); and finally, how 
to interpret field test data in terms of the importarjce and comprehensiveness of 
ob j ect I ves . 

The latter problem. Interpreting field test data,. has become a particular- 



Competency-Based 
k 

ly Irksome but integral part of the life of any de\?eloper concerned with . : 
competency- based measurement. For Instance, consider a developer faced with 
a situation In which all students correctly answer" all questions on a compe- 
tency-based pretest. In theory, this would suggest that the pretest was too 
easy and needed revision or that the objectives were Irrelevant since they 
had already been achieved. But wait, and consider another developer confronted 
with a situation In which all students do poorly on the competency-based post- 
test. In theory, this would suggest that the , mater lal s used were In need of 
revision since they were not teaching successfully, that the posttests were 
too hard, or that the objectives were Inappropriate for the learners who were 
Incapable of achieving them. 

Unfortunately for both developers, it Is often Impossible to decide which 
Interpretation' Is correct, that Is, whether the £est Is at fault In being 
. too easy or too hard or whether the materials ne^d revision. This ambiguity 
exists because the measures used to test the efficiency of the Instructional 
product typically have not been validated themselves. In these situations 
the two. developers' data are truly unlnterpretable, but what If they are all 
the developer has? There are very few Instances In which developers have 
access to validated Instruments that yield credible Information. In fact. 
In most cases, product development and test development occur together, with 
data from one rough draft used In an Intuitive way to refine the other, a 
circumstance closely resembling the inauspicious one In which the blind lead 
the bl Ind. 

Given the problems -In formulating objectives and In Interpreting informa- 
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fclon about them, not to mention the costs and effort' involved In field 
tests, what should the concerned developer do?. Unfortunately, at present, 
few rules are available although attempts have beenmade to describe de- 
velopment activities and to provide general guidelines for them (Popham & 
Baker, 1971). 

Formulating Items for Cofnpetency-Based Measures 

Once objectives have. been formulated, the developer Is then confront- 
ed with the task of constructing items to measure achievement. This task 
Is a very difficult one for developers who are frequently individuals 
with lif.tle formal training in test writing and validation, and who may 
be skillful instructors, but poor item writers and ^n^lysts^ fupttfer^ 
even having knowledge of psychometrics does not vitiate many unanswered 
questions about competency-based measurement that seem to exist to com- 
plicate the developer's jco. For example, how many Items are needed to 
measure an objective? Even highly specific objectives have potential 
Item pools of well over several thousand items (Bormuth, 1970; HIvely,. 
1970; et. al, 1973)» Should the developer sample items? . If so, how? A 
related quescJon concerns the selection of Items according to their dif- 
ficulty. Thus, the developer must not only decide how many Items are 
needed to measure an objective, but also determine the level of difficulty 
for each item. Given the problems involved in test generation, the de- 
veloper would be well advised to call in experts or to obtain *Mtem forms*' 
or "shells" to guide test development (Cronbach, 1971; Hiyely et al, 1973} 
Skager, 1973) • However, experts are expensive and currently available 
I tem-wrl ting rules are far from generally available or even applicable to 
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many competency-based instructional development concerns.. 
Score interpretation 

Competency-based measurement provides descriptions of what indivi- 
duals I<now or are likely to be able to do and reports scores In terms of 
performance standards. A major problem often arises, however, in establish- 
ing the meaningfulness of the standards since to be legitimate and cred- 
Ib^e, they should preferably be empirically justifiable. For example, 
a score of 7 out of 10 Items will only have meaning If systematic study 
has shown that those who receive the score can actually do- something that 
others who have not reached this level cannot do. Needless to say, such 
systematic investigation is costly and requires ability to conduct test- 
ing research. Few instructional prod"uct developers have had the oppor- 
tunity to participate in this kind of research, ancf few development bud- 
gets ir^clude provisions for it. In its absence, many developers must; 
continue to unhappily rely on arbritrary scores like 7 out of 10 to 
estimate gains in competence for information about their product. 
E st<<bl Ishing the Technical Excellence of Competency-Based Measurement 

It is axiomatic that all tests and measures must be field tested be- 
fore decisions can be based on them. Thus, when the construction of an 
achievement test's items is complete, the test must be validated. The 
purpose of the validation is to ensure the quality of each item, and the 
test as a whole, and to be able to declare the test to be a valid 
and a reliable instrument. However, developers are usually not trained 
In the vicissitudes of conducting and costing test validation. To furthn- 
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complicate an already difficult task, debate confnues over t!,e appropri- 
ateness of classical Indexes of reliability and validity, which are based 
on variance, for Instruments that are used In situations that produce 
Uttle variation in scores (Popham & Husek, 1969). Such situations are 
well known to Instructional product developers who will undoubtedly be 
familiar with the following occurrence: Before Instruction, none of the 
students are competent with respect to the objectives and they all receive 
low scores on the pretest, whereas after Instruction, they all receive 
high scores. Most theoreticians would argue that this Is the Individual 
or idiosyncratic case, that variation is Inevitable In most instructional 
situations, and that therefore, classical Indexes are appropriate (Harris. 
1373). Nonetheless, as Interesting and Important as the issue may be, 
developers cannot be expected to take sides in the debate and are to be 
forgiven if they regard data from competency-based measurement with a 
degree of confusion and distrust. 
Use of Competency-b ased Measurements; Conclusions 

At present, expertise for developing reliable and yalid competency- 
based Instruments Is not available in a form that an Instructional develop- 
er can readily translate Into practice. The formulation of objectives, , 
construction of items. Interpretation of scores, and determination of re- ' 
liaoility and validity remain problematical. Thus, data obtained from 
competency-based measurement, which are to be used to improve products 
and describe performance, must be viewed with extreme care, if not actual 
suspicion. The developer should take care to confirm any findings from 
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competency-based measurement by relyfng upon add5tIonal sources of Infor- 
mation Uke students' comments and observers' ratings. Further, develope 
should become acutely aware of the part Instrument development is likely 
to play In product design, and be ready to acquire psychometric expertise 
to prepare them for the problems that are likely to energe, fn the 
me^ntlne, data derived from cmpetency-based measurement should be handled 
with care and at the developer's own risk. 
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Footnote • • 

' Competency-based achievement tests and erf tcrlon-referenced tests are 
Identical In most respects. The major difference between them Is that compe- 
tency-based tests are always di rectly job-rplated, while cH terion-referenoed 
tests can also be used to measure academic skills that may be only Indirectly 
related to Job performance. Thus, competency-based tests are really a type 
of criterion-referenced testing. 
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